Skip to content

YARN-11824: Prevent test dependencies from leaking into the distro.#7885

Closed
cnauroth wants to merge 1 commit intoapache:trunkfrom
cnauroth:cnauroth-YARN-11824
Closed

YARN-11824: Prevent test dependencies from leaking into the distro.#7885
cnauroth wants to merge 1 commit intoapache:trunkfrom
cnauroth:cnauroth-YARN-11824

Conversation

@cnauroth
Copy link
Contributor

@cnauroth cnauroth commented Aug 19, 2025

Description of PR

First, make sure test libraries like jersey-test-framework-provider-jetty are limited to scope test.

Then, update other modules that didn't know they were getting their test dependencies through YARN to depend on test libraries explicitly.

How was this patch tested?

I used mvn dependency:tree to check that the test dependencies were gone after these changes. Before the changes, we had this in the final distro after mvn clean package -Pdist -Dtar -DskipTests -DskipShade:

/tmp/distro/hadoop-3.5.0-SNAPSHOT/share/hadoop/mapreduce/lib/mockito-junit-jupiter-4.11.0.jar
/tmp/distro/hadoop-3.5.0-SNAPSHOT/share/hadoop/yarn/lib/junit-jupiter-5.11.0.jar
/tmp/distro/hadoop-3.5.0-SNAPSHOT/share/hadoop/tools/lib/junit-4.13.2.jar
/tmp/distro/hadoop-3.5.0-SNAPSHOT/share/hadoop/tools/lib/junit-jupiter-engine-5.13.3.jar
/tmp/distro/hadoop-3.5.0-SNAPSHOT/share/hadoop/tools/lib/junit-platform-commons-1.13.3.jar
/tmp/distro/hadoop-3.5.0-SNAPSHOT/share/hadoop/tools/lib/junit-platform-engine-1.13.3.jar
/tmp/distro/hadoop-3.5.0-SNAPSHOT/share/hadoop/tools/lib/junit-jupiter-params-5.13.3.jar
/tmp/distro/hadoop-3.5.0-SNAPSHOT/share/hadoop/tools/lib/junit-jupiter-api-5.13.3.jar
/tmp/distro/hadoop-3.5.0-SNAPSHOT/share/hadoop/yarn/lib/hamcrest-3.0.jar
/tmp/distro/hadoop-3.5.0-SNAPSHOT/share/hadoop/tools/lib/hamcrest-core-1.3.jar

After these changes, everything is gone except for the stuff in hadoop/tools/lib. This is apparently intentional dependencies of hadoop-tools/hadoop-compat-bench, but it won't creep into the classpath for dependent projects like Parquet.

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

@cnauroth
Copy link
Contributor Author

cnauroth commented Aug 19, 2025

@steveloughran and @slfan1989 , review please? Thank you!

pan3793
pan3793 approved these changes Aug 19, 2025
View reviewed changes
Copy link
Member

@pan3793 pan3793 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also spotted this issue today, thank you for fixing it!

@slfan1989
Copy link
Contributor

slfan1989 commented Aug 19, 2025

@steveloughran and @slfan1989 , review please? Thank you!

@cnauroth Many thanks for spotting and fixing this issue. There are still some follow-up issues with the Jersey upgrade that we’ll need to resolve step by step.

@hadoop-yetus
Copy link

hadoop-yetus commented Aug 19, 2025

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 31s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 xmllint 0m 0s xmllint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+0 🆗 mvndep 8m 53s Maven dependency ordering for branch
+1 💚 mvninstall 33m 7s trunk passed
+1 💚 compile 16m 1s trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚 compile 14m 1s trunk passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚 mvnsite 4m 20s trunk passed
+1 💚 javadoc 4m 0s trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚 javadoc 4m 3s trunk passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚 shadedclient 120m 18s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 34s Maven dependency ordering for patch
+1 💚 mvninstall 2m 9s the patch passed
+1 💚 compile 14m 54s the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚 javac 14m 54s the patch passed
+1 💚 compile 13m 36s the patch passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚 javac 13m 36s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 4m 18s the patch passed
+1 💚 javadoc 4m 12s the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚 javadoc 4m 2s the patch passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚 shadedclient 50m 52s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 6m 7s hadoop-yarn-common in the patch passed.
+1 💚 unit 4m 19s hadoop-yarn-server-applicationhistoryservice in the patch passed.
+1 💚 unit 9m 15s hadoop-mapreduce-client-app in the patch passed.
+1 💚 unit 22m 39s hadoop-yarn-applications-distributedshell in the patch passed.
+1 💚 unit 21m 10s hadoop-yarn-services-core in the patch passed.
+1 💚 asflicense 1m 22s The patch does not generate ASF License warnings.
269m 45s
Subsystem Report/Notes
Docker ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7885/1/artifact/out/Dockerfile
GITHUB PR #7885
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint
uname Linux f3a628b010ba 5.15.0-140-generic #150-Ubuntu SMP Sat Apr 12 06:00:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / e5a8e3f
Default Java Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7885/1/testReport/
Max. process+thread count 876 (vs. ulimit of 5500)
modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7885/1/console
versions git=2.25.1 maven=3.6.3
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran
steveloughran approved these changes Aug 19, 2025
View reviewed changes
Copy link
Contributor

@steveloughran steveloughran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@cnauroth cnauroth closed this in afe2aa9 Aug 19, 2025
@cnauroth
Copy link
Contributor Author

cnauroth commented Aug 19, 2025

Thanks for the reviews, @pan3793 , @slfan1989 and @steveloughran ! I committed this to trunk.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@steveloughran steveloughran steveloughran approved these changes

@slfan1989 slfan1989 slfan1989 approved these changes

+1 more reviewer

@pan3793 pan3793 pan3793 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

build MapReduce trunk YARN

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@cnauroth @slfan1989 @hadoop-yetus @steveloughran @pan3793